fix(indexer): UTF-8 encoding for git log on non-UTF-8 Windows locales#143
Open
rajkumarsakthivel wants to merge 6 commits into
Open
fix(indexer): UTF-8 encoding for git log on non-UTF-8 Windows locales#143rajkumarsakthivel wants to merge 6 commits into
rajkumarsakthivel wants to merge 6 commits into
Conversation
#140) On Windows with non-UTF-8 system locales (e.g. Korean cp949), Python's subprocess.run with text=True decodes using the system codepage. Git log output is UTF-8 regardless of locale, so non-ASCII commit messages or author names cause UnicodeDecodeError, aborting git history indexing. Pass encoding="utf-8", errors="replace" on both subprocess.run calls in index_commits to match git's actual output encoding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #140. Git history indexing fails with
UnicodeDecodeErroron Windows machines using non-UTF-8 system locales (e.g. Korean cp949, Japanese Shift_JIS, Chinese GBK).The two
subprocess.runcalls inindex_commitsusedtext=Truewithout an explicitencoding=. Python decodes subprocess output usinglocale.getpreferredencoding(False), which on Windows returns the system ANSI codepage. Since git log output is always UTF-8, any non-ASCII commit message or author name that isn't valid in the system codepage raisesUnicodeDecodeError.Fix: pass
encoding="utf-8", errors="replace"on both calls, matching git's actual output encoding. Theerrors="replace"fallback handles the rare case where git output contains non-UTF-8 bytes (e.g. legacy repos with Latin-1 commit messages) by substitutingU+FFFDinstead of crashing.One-line change per call site, no new dependencies.